The present invention relates generally to interface systems for allowing humans to interface naturally with computer systems and simulations, and, more particularly, to interface systems that allow a user to interact with computer simulated environments both visually and through haptic sensations.
Computer systems are used increasingly to provide simulated experiences to users, for purposes such as training, remote control and sensing, and entertainment. These systems typically include a visual display, such as a standard video monitor, through which visual information generated by the simulation is presented to the user. Virtual reality systems may also include more sophisticated visual displays, such as a head-mounted displays, which aid the user in becoming xe2x80x9cimmersedxe2x80x9d into the simulated environment by attempting to remove any visual input to the user that is not being generated by the simulation. To further enhance the user""s simulated experience, sound is provided through speakers or headphones that are connected to the computer system and provide aural information that is controlled by the simulation. In addition, interface devices commonly allow users to interact with the simulation through manual commands or gestures made by the user, i.e., the device tracks the user""s xe2x80x9ckinestheticxe2x80x9d activities. Keyboards, trackballs, mice, joysticks, pedals, steering wheels, and joypads are interface devices that allow human interaction with computer systems.
Typical human interface devices are input only: They track a users physical activities, but do not provide a means of presenting physical information back to the user. For example, a traditional computer joystick will allow a user to manipulate an object within a computer simulated environment. However, if that object encounters a simulated obstruction, the interaction will only be experienced visually (and maybe aurally through sound feedback), not physically. In other words, when a user manipulates a joystick and causes a computer generated object to encounter a computer generated obstruction, the user will not feel the physical sensation of one object hitting another object. Thus, the user is not truly xe2x80x9cimmersedxe2x80x9d in the simulation as the user receives no physical feedback in conjunction with simulated experiences.
This missing feedback modality can be supplied by a force feedback interface device. A force feedback human interface device is a special class of manual human interface that not only tracks a user""s manual gestures but also includes means of presenting physical sensations back to the user. A force feedback device typically includes sensors for tracking a user""s motions and actuators for producing physical forces representative of the simulated interactions. Using a force feedback joystick, a user may manipulate a simulated object in a simulated environment such that if that object encounters forces or other simulated phenomena, the actuators in the device are commanded to simulate a sensation associated with the interaction. For example, if a user manipulates a force feedback joystick to control a simulated brick and causes the brick to contact a simulated piece of metal, the computer would generate forces on the joystick so that the user would feel a physical sensation representative of the encounter. Such physical feedback makes the computer-simulated environment significantly more realistic to the user.
One goal in developing realistic computer simulated environments is to allow users to take advantage of their natural manual dexterity and basic physical skills when interacting with computer simulations. When trying to create a simulated environment in which users can make use of their dexterous skills, it is important to establish a meaningful and intuitive correlation between the information displayed visually and the manual information perceived as xe2x80x9cfeelxe2x80x9d through the force feedback interface device. This is particularly important when trying to create a simulation environment for allowing users to engage in computer-simulated xe2x80x9csportingxe2x80x9d interactions or similar simulations which are receptive to a wide range of physical skill in the user.
In such applications, a meaningful and intuitive correlation between the visual display of sporting events and manual physical interactions required of that sporting event is critical.
For example, in a simulated sporting environment, a user might wield an interface device which represents a paddle or racquet within the simulated environment. The user will manipulate the simulated paddle and interact with other simulated entities such as pucks, balls, walls, barriers, and even additional paddles manipulated by other players/users of the environment. The interaction between the user""s paddle and other simulated entities in the environment will be displayed visually as well as physically. When the user moves the simulated paddle, the user""s kinesthetic sensation of paddle location must be reasonably correlated to the representation of paddle location as displayed visually. If the user perceives kinesthetically that his hand has moved to a given location but views a non-corresponding change in visual location, the realism will suffer and the user will be unable to take advantage of his or her full dexterous skills. In some cases, an unnatural correlation between visual and physical experiences will make it impossible for the user to execute the simulated sporting task.
When there are force feedback sensations provided to the user, this correlation between visual and physical becomes even more important. If the user moves a paddle and feels the sensation of the paddle hitting a simulated puck at a given location, but views the paddle-puck interaction at a different location, the realism will suffer and the user will be unable to take advantage of his/her full dexterous skills. Thus while force feedback is intended to increase the realism of a computer simulated environment and enable dexterous manual activities, if done incorrectly, force feedback can disrupt, confuse, and even inhibit a users ability to take advantage of his or her natural dexterous skills.
What is needed therefore is a computer system providing both visual display and force feedback interfacing mechanisms that can establish a natural and meaningful correlation between information displayed visually and physical interactions perceived manually. Unfortunately, there are limitations to force feedback interface devices which make it difficult to represent many simulated physical interactions. For example, a force feedback device has cost, size, and safety constraints which limit the maximum force output that can be applied to a user and therefore make it infeasible to generate sensations corresponding to general interactions between rigid surfaces. Thus, a user may encounter a simulated hard surface, but the user will be able to easily overpower the resistance because of such force output magnitude limitations. However, it is very easy to visually display a depiction of interactions between rigid surfaces which represents a rigid and impenetrable barrier. This dichotomy between the limitations of visual display and physical display must be resolved, especially in simulated sporting interactions where physical skill is central to the simulation. Therefore, there is needed methods for allowing visual display of simulated interactions and physical display of simulated interactions to deviate from their natural mapping at instances when the force feedback device is simply incapable of representing physical interactions which can be represented visually. Such methods must be developed so as not to greatly disrupt a users ability to use his/her natural manual dexterity.
The present invention is directed to controlling and providing force feedback to a user operating a human/computer interface device in conjunction with a simulated environment implemented by a host computer system. The user views graphical images on a display while feeling realistic force sensations using safe, practical force feedback devices such that the user is involved in an immersive and intuitive simulation.
More specifically, the present invention provides a method and apparatus for providing a computer-simulated environment visually displaying simulated representations and providing force feedback sensations to a user who controls one or more of the simulated representations using a force feedback interface device. In a preferred embodiment, the simulation is of a sporting environment in which one or more users can compete in physically challenging manual tasks which require dexterity and skill. To provide a realistic sporting environment and to enable the user to provide dexterous control of the simulation, the visually displayed representations are naturally correlated with the manual motions and force feedback experienced by the user. Thus, the manual motions of the user (input) and force feedback (output) are naturally correlated to interactions of user-controlled simulated objects with other displayed simulated objects.
According to one embodiment of the method of the invention, the position of a simulated object generated within a computer simulation is controlled by a user according to a position control mapping. Such a mapping indicates that a change in position of a physical object grasped and moved by the user is directly mapped to a corresponding change in position of the displayed user-controlled simulated object in the simulation. A game apparatus of the present invention allows a position control mapping to be implemented. One embodiment provides a console having two opposing display screens which two players can operate to compete in a sporting simulation. Other embodiments of the game apparatus include a single display screen tabletop, overhead projector displays, and displays projected from beneath a table surface. The interface device of the game apparatus includes a handle for the user to grasp and move in the two degrees of freedom of a plane. Another embodiment conceals a player""s hands from the player""s view to allow a more immersive experience. In yet another embodiment, a racquet interface device having a racquet handle moveable in two or more degrees of freedom and a sensor stage allows a player to interact with a sporting simulation.
An embodiment of the present invention for a sporting simulation includes a paddle simulated object controlled by a player and a ball or puck which interacts with the paddle. The player can skillfully move the paddle to interact with the ball and feel forces on the paddle as if the ball and paddle have mass and other physical characteristics. The paddle is preferably compliant and can be used as a xe2x80x9cslingxe2x80x9d which can catch the ball and be used to direct the ball in a desired direction influenced by the player""s skill. One embodiment allows the player to xe2x80x9ctrapxe2x80x9d the ball when it is engaged with the paddle by the use of an input device such as a button. Obstruction objects such as walls can also be included in the simulation and displayed to interact with the ball and the paddle.
The present invention also provides a method and apparatus for providing realistic force feedback sensations in a simulation despite limitations to force feedback devices. More specifically, the mapping between the position of the user-controlled simulated object and the position of the physical user object is broken under specific circumstances to provide a realistic force interaction when forces output to the user are limited in magnitude due to safety, cost, and size concerns. Thus, for example, when the user-controlled simulated object collides with another simulated object, such as a wall, the simulated object is visually blocked; however, the physical user object may still be moved xe2x80x9cintoxe2x80x9d the wall. This breaking of the mapping is associated with force feedback that is effective to impart a physical sensation corresponding to the interaction of the simulated objects.
In one embodiment, the force feedback corresponds to a restoring force that is proportional to the magnitude of the breaking of the simulation. A more specific embodiment incorporates a spring force of the form F=kx as the restoring force, where F is said restoring force, x is the magnitude of the displacement of the graphic object in the absence of the simulated interaction, and k is a spring constant parameter. Another embodiment incorporates both spring and damping forces.
In another embodiment, the second simulated object is controlled by a second user. Variants of this embodiment include those for which the restoring force is a spring force of the form F=k(x1+x2), where F is the restoring force, x1 and x2 are the magnitudes of the displacements of in the absence of the simulated interaction, and k is a spring constant parameter. Another embodiment incorporates both spring and damping forces. In another variant, the restoring force includes a weighting factor to allow variation in the paddles of users in the ability to block or push other player""s paddles.
In yet other embodiments, multiple players can interact in a simulation such that several paddles and/or balls are provided and influence forces on other users if the paddles interact. The multiple users can interact in a simulation implemented by a single computer system, or in a simulation implemented by multiple computer systems that are linked over a network.
The simulated environment of the present invention allows a player to realistically and skillfully interact with a computer-generated simulation. The position control paddle-ball simulations allow a player to exercise a great degree of skill and dexterity when competing against a computer opponent or other users in a sporting simulation, thus providing a highly entertaining and enjoyable activity. The multi-player sporting simulations allow several players to interactively compete at the local or remote computer sites, thereby providing multiple human components to the competition. The breaking of the position control mapping and provision of restoring forces allows a user to feel realistic collision and interaction forces despite using a force feedback device having force magnitude limitations due to safety, cost, and weight constraints. The interface devices of the present invention, allowing natural motions of the user manipulating the interface device, are ideally suited to allow the user to skillfully interact in sporting simulations.
These and other advantages of the present invention will become apparent to those skilled in the art upon a reading of the following specification of the invention and a study of the several figures of the drawing.